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Abstract 



The overarching goal of the program is to empower staff members from nonprofits to design 
evaluations that help further their organizational mission, thereby addressing the needs of their clients 
more effectively and efficiently. During 2001-02 we offered five bi-monthly two-day sessions. 

Twenty participants from ten nonprofits in the region attended these sessions. Program topics include 
utilization-focused evaluation, the program’s theory of action, evaluation designs, assessment methods, 
data analysis, and reporting results. While enrolled in this yearlong program, all participants 
conducted a wide range of evaluation activities in their organizations drawing upon what they had 
learned. At each session they shared their progress with fellow participants, provided feedback to each 
other, and sought instructors’ guidance. This active and collaborative learning allowed them to 
develop a better understanding of key concepts, principles, and methods in the context of their work. 
They gave uniformly high ratings to the program (4.5 to 5.0 on a five-point scale.) These high ratings 
were confirmed by their enthusiasm and engagement throughout the program, by the high rate of 
attendance at each session, by the positive tone of responses to the open-ended questions, and by their 
openness to share their work with each other. 
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Teaching Program Assessment to Adult Learners 

Chandra M. Mehrotra, Ph.D. 

The College of St. Scholastica 

Evaluation and assessment have become increasingly important over the past decade. Expanding 
the movement that first began in the educational arena, documented outcomes are now also expected 
from the business world, all levels of federal and state government, and non-profit community 
agencies. Accrediting associations, shareholders, and philanthropic agencies alike are demanding 
increased accountability from the organizations with which they work. 

While these expectations span geographic boundaries, they present a particular challenge for small 
agencies outside of metropolitan areas. Professionals from such agencies often have limited 
knowledge of successful practices in other communities, infrequent contact with experts in program 
design and assessment, few opportunities to develop plans in collaboration with others, and insufficient 
feedback from colleagues with diverse perspectives and backgrounds. These factors limit their ability 
to initiate programs that respond appropriately to community problems. 

Given these critical needs, we have designed and implemented a yearlong training program for 
staff members from a wide range of community agencies in Northeastern Minnesota. The overarching 
goal of the program is to empower nonprofit agency staff to design evaluations that help further their 
organizational mission, thereby addressing the needs of their clients more effectively and efficiently. 
Specific objectives include: 

1. Strengthen the capacity of organizations to address the needs of their clients through staff 
engagement in hands-on program evaluation. 

2. Increase participants’ knowledge, skills, and motivation for conducting evaluations aimed 
at improving program effectiveness and furthering organizational mission. 

3. Infuse evaluative thinking in the region by fostering networking relationships among 
representatives from diverse organizations. 

Premises 

The program design is based on the following premises which have been informed by our work 
with an NIA-supported research training program in aging for psychology faculty (Mehrotra, 2003), as 
well as our earlier experience with NSF-funded program on faculty enhancement (Mehrotra, 1996). 

1. Given the increased pressure from funding agencies, staff members from nonprofit agencies are 
ready to strengthen their knowledge of program evaluation and apply it in their work settings. 
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2. Effective training programs engage participants in active and collaborative learning 
(McKeachie, 1999). Participants learn key concepts and principles in the context of their 
projects, share their progress, provide feedback to each other, and seek instructors’ guidance. 

3. A one-time workshop is not sufficient. Sustaining interaction with program faculty and with 
colleagues from other agencies and active involvement in follow-up activities accelerates 
participants’ progress toward the goal of conducting useful evaluations (Mehrotra, 2002). In 
addition, the more time participants are engaged in learning, the greater the amount of what 
they learn and how they use it (Chickering & Gamson, 1991). 

4. Training in program evaluation should include a discussion of proposal evaluation, evaluation 
management, and personnel and product evaluation. A widening of participants’ perspective 
helps them to avoid reinvention of the wheel and omission of relevant aspects of what they are 
supposed to be evaluating (Scriven, 1993). 

5. Program evaluation contributes to enlightened discussion of alternative ways for achieving the 
intended outcomes (Cronbach & Associates, 1980; Posavac & Carey, 2003). Training 
activities should not only focus on how to monitor program implementation and assess its 
impact but also on how to conceptualize the theory upon which program is based. 

6. Adult learners appreciate the opportunity for reflective practice (Schoen, 1987). The reflective 
practice process involves (a) identifying an issue, interest, or concern; (b) agreeing to try 
something; (c) observing what is tried; (d) reporting back to the group individually; (e) 
identifying patterns of experience or themes across reports; (f) deciding what to try next in light 
of the findings; and (g) repeating the process with new commitment to action (Patton, 1997). 

Program Design 

During 2001-2002, twenty participants met bi-monthly for an intensive two-day session. The 
group included two participants from each of the ten community agencies selected for the first year of 
the program. Examples of agencies represented in the first cohort include: Area Agency on Aging, 
Native American Residential Treatment Center, Mental Health Center, Center Against Sexual and 
Domestic Abuse, and Community Action Program. Program sessions included topics such as 
utilization-focused evaluation, the program’s theory of action, logic models, evaluation design, 
assessment methods, data analysis, and reporting results. At each session the participants also devoted 
substantial amount of time to discussing the projects they are currently evaluating. This engaged them 
in practical application of what they were learning and allowed them to give feedback to each other, to 
obtain instructors’ guidance, and to learn relevant concepts, principles, and approaches in the context 
of their project. 
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To illustrate effective methods of data collection, useful approaches to developing logic models, 
and utilization-focused evaluation reports, the instructors provided the participants copies of 
questionnaires, logic models, and evaluation reports. Reviewing these materials stimulated the 
participants to discuss distinctive features of effective models, measures, and reports and provided 
them with concrete examples that they can use in their work settings. In addition, the instructors often 
used examples to illustrate approaches that should not be used in evaluation projects. In other words, 
the instructors used a variety of instructional materials and strategies to promote active and 
collaborative learning throughout the program. Many participants shared program handouts with 
colleagues within their organizations. Such sharing often led to additional discussions, thereby 
fostering increased understanding of the topic they were reviewing. Table 1 presents the topics 
covered during the first year of the program. 



Table 1 

Program Sessions Offered in 2001-2002 



Date 


Session 


Activities 


November 1 & 2, 
2001 


2-day opening 
retreat 


Tonics: Orientation to the program, its outcomes, 
design, expectations, and procedures. Overview of 
Utilization-Focused Evaluation. 
Hands-on-Activity: Participants introduce 
themselves, the agencies they represent, and the 
project they are currently evaluating. 


December 1 9 & 20, 
2001 


2-day session 


Topic: The program’s theory of action. 

Hands-on Activity: Participants discuss the logic 
model they may use for their project; invite fellow 
participants and the instructors to give feedback 
and suggestions. 


January 30 & 31, 
2002 


2-day session 


Tonic: Data collection methods 
Hands-on Activity: Participants share their data 
collection methods with each other and with the 
instructor for comments and suggestions. 


March 6 & 7, 
2002 


2-day session 


Tonic: Data analysis and interpretation. 
Hands-on Activity: Participants share examples 
of how they have conducted data analysis, 
presented key findings, and prepared evaluation 
reports. 


April 17 & 18, 
2002 


2-day session 


Topic: Reporting evaluation findings and utilizing 
them for program improvement. 

Hands-on Activity: Participants conduct a content 
analysis of reports on what they had learned in 
each session. 
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Program Faculty 

All sessions were facilitated by the author in collaboration with Michael Quinn Patton. Patton is 
the author of five books on program evaluation including Utilization-Focused Evaluation (Patton, 
1997) and Qualitative Research & Evaluation Methods (Patton, 2002). We made copies of both of 
these books available to participants by sharing their cost. 

Program Outcomes 

Throughout the program all participants conducted a range of evaluation activities directly related 
to their work. They learned how to develop a logic model for their programs, how to design an 
evaluation questionnaire, how to conduct data analysis for their projects, and how to present the results 
to promote utilization. They brought the drafts of their models, questionnaires, data analysis, and 
reports to the institute sessions for feedback from fellow participants and from the instructors. This 
allowed them to (a) clarify their thinking in the context of their work; (b) test their ideas; (c) identify 
themes or patterns across reports; (d) make improvements in their initial drafts; and (e) decide what to 
try next in light of the findings and discussions (Schoen, 1987; Patton, 1997). Participants appreciated 
such opportunities for reflective practice throughout the year. 

Program Evaluation 

With the goal of engaging the participants in reflective practice and obtaining their feedback on a 
regular basis, the author asked them to respond to three questions at the end of each session: 

• What are the five things that you are taking away from this session? 

• What changes would you suggest to strengthen the program? 

• How would you rate the usefulness of the session on a five-point scale? 

This approach was highly effective. It gave the participants an opportunity to (a) reflect upon what 
they had learned and describe it in layman’s language; (b) provide suggestions for program 
improvement; and (c) assess the usefulness of what they learned in the session. The author prepared a 
summary of their responses for each session and shared it with them at the following session. In 
addition, he examined their suggestions and used them to make modifications and improvements in the 
program. As a concluding exercise at the last session, the participants worked in small groups to 
conduct a content analysis of the responses they had given for each session. This exercise allowed 
them to process what they had learned in the program as a whole, to identify the themes in what they 
had reported learning in each session, and to discuss its implications for continued learning. The 
content analysis indicated that participants had learned: 

• to identify the intended users before conducting the evaluation. 

• to distinguish between formative and summative evaluation. 

• to construct a logic model to guide program development and improvement. 
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• to seek buy-in from colleagues in the organization. 

• to learn from each other. 

• to use reflective practice in team settings. 

• to design evaluation instruments. 

• to use pretest data to help define emphasis of program content. 

• to distinguish between reliability and validity. 

• to use multiple methods of data collection. 

• to appreciate the value of qualitative methods. 

• to employ a variety of sampling techniques. 

• to use strategies that yield high response rates. 

• to be careful not to ask personal information that is not needed. 

• to present findings aimed at intended users. 

On a five-point scale, where l=poor and 5=excellent, the mean rating for each session ranged 
between 4.50 and 5.00. These consistently favorable ratings were confirmed by the continuing high 
rate of attendance at each session, by the responses to the open-ended questions, and by the highly 
positive feedback the participants provided throughout the program. 

Conclusion 

Given the isolation they experience and the lack of training opportunities available to them, the 
participants were hungry for new knowledge and were eager to develop essential skills in assessment 
and evaluation. One indication of their motivation was the high rate of attendance throughout the year. 
Another indication was their willingness to participate in group discussions, to share the work they 
were doing on their projects, and to seek consultation to improve evaluation quality. It was clear that 
they had formed a community of learners that was open to new ideas and approaches useful in 
evaluating projects their agencies undertake. 

Designing and offering this program has been a highly satisfying experience for me. It has allowed 
me to collaborate with Michael Patton, a distinguished scholar in program evaluation. It has given me 
an opportunity to address a pressing community need and to foster continuing partnerships between the 
College and the nonprofits. I have developed a better understanding of what nonprofits serve the 
region, what they try to accomplish, what difficulties they face, and how assessment and evaluation 
can be of help to them. In light of favorable results achieved with the program during the first year, we 
offered it to a second group of nonprofits during 2002-03 and achieved similar outcomes. Now we are 
planning to make the training available in another region of the state. 
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